In order to solve the problem that it is difficult to leverage the performances of effect and efficiency, an image matching algorithm based on the Histogram of Gradient Angle (HGA) was proposed. After obtaining the key points by Features from Accelerated Segment Test (FAST), the block gradient and the new structure as dartboards were introduced to descript the local structure feature. The image matching algorithm based on HGA can work against the rotation, blur and luminance and overcome the affine partly. The experimental results, compared with Speeded Up Robust Feature (SURF), Scale Invariant Feature Transform (SIFT) and ORB (Oriented FAST and Rotated Binary Robust Independent Elementary Features (BRIEF)) in the complex scenes, demonstrate that the performance of HGA is better than other descriptors. Additionally, HGA achieves an accuracy of over 94.5% with only 1/3 of the time consumption of SIFT.